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In this study, 7th grade students were observed completing a 
series of lessons in an online science course to explore their 
thinking and strategies for: (1) using curriculum resources 
for learning, (2) developing thinking and strategies for self- 
assessing “assessment readiness”, and (3) exploring the rela¬ 
tionship between resource use and student grade in science, 
end-of-lesson assessment score, and student learning style. A 
convergent-parallel mixed methods research design was used 
to collect both qualitative and quantitative data in the same 
phase of this study, in order to merge the results and gain a 
greater understanding of student learning in the online envi¬ 
ronment. Observations of students completing lessons online, 
survey, and interview data were collected. Qualitative results 
indicate that students demonstrated various levels of meta- 
cognitive strategy use in choosing online curriculum resourc¬ 
es and in determining their readiness for lesson assessment. 
Quantitative results indicated that resource use was higher 
for multi-media learning resources and lower for notes-based 
resources. Use of at least 70% of lesson resources overall 
showed a relationship with a higher student grade in science 
and higher end-of-lesson assessment score. Student learning 
style showed mixed results related to resource use. Contin¬ 
ued mixed-methods research observing student learning in 
the online environment is needed to better understand the in¬ 
dividual student experience of learning online, and to inform 
specific interventions within the online learning environment 
to support learning as students transition into more effective 
self-directed learners. 
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INTRODUCTION 

Online and distance environments can support effective student learning 
in many ways. The purpose of this study was to observe students complet¬ 
ing a series of lessons in the online learning environment in order to explore 
what can be learned about how students are using online lesson resources 
for learning when they are choosing them as self-directed learners. 

LITERATURE REVIEW 
Support & Engagement Factors 

Cavanaugh and Blomeyer (2007) summarized student characteristics, 
course design factors, and instructional factors that impact K-12 student 
success with online learning. Course design factors included a structured 
course design with clear expectations and study guides, study-skill devel¬ 
opment, models of portfolios, meaningful curriculum, and opportunities to 
collaborate with peers. Instructional factors included guidance for learning 
in the virtual setting and for use of online tools, and frequent and timely 
response to student questions. Ongoing communication and feedback were 
deemed especially helpful for students who had learning disabilities and 
attention deficits. Student characteristics related to success included self- 
discipline and self-motivation, having developed effective strategies as 
learners in traditional educational settings, and learning styles that include 
abstract conceptualization and reflective observation. Gladwell (2000) used 
the success of the Sesame Street television show in teaching literacy to pre¬ 
school children through the distance learning media of television to explain 
that in order for something (learning) to “stick” (be memorable), the mes¬ 
senger (format of the message) and the message (content) had to be appeal¬ 
ing to the learner so that the student’s attention was held long enough to 
learn the material. When the message was clear, preschoolers attended to 
the information even with toys in the room competing for their attention, 
but if preschoolers couldn’t make sense of the message they turned their at¬ 
tention to something else (Gladwell, 2000). For effective learning to “stick” 
with students, students needed an environment in which to practice using 
what they were learning across many contexts, get timely feedback on their 
learning process and progress, and “have time to reflect on the feedback 
they receive, make adjustments, and try again” (Project 2061, 2012, p.l). 
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Scaffolding and Support Strategies 

Bransford, Brown, Cocking, Donovan, and Pelligrino (2000) explained 
several key shifts in the learning sciences related to student learning: the 
meaning of “knowing” has shifted from being able to remember and repeat 
information to being able to find and use it” (p. 5). Especially in helping 
students take control of their own learning, students “must learn to recog¬ 
nize when they understand, and when they need more information,” a pro¬ 
cess supported by metacognition or the ability of students to “predict their 
performances on various tasks and to monitor their current levels of mastery 
and understanding” (p. 9). Shannon (2008) studied the effects of teaching 
high school chemistry on students’ metacognitive strategies and assessing 
their own learning style. His results indicated that there was a significant 
relationship between student use of metacognitive strategies, student learn¬ 
ing, and self-assessment behaviors. Wang, Peng, Huang, Hou, and Wang 
(2008) reported that the level of learning strategy is one of the most im¬ 
portant factors impacting learning outcomes, followed by learner motivation 
and willingness to apply effort to learning. 

Research has shown a number of benefits of virtual learning for students 
choosing online learning options. Freeman and Crawford (2008) reported 
success with online learning providing scaffolding of subject matter instruc¬ 
tion for English as Second Language (ESL) learners. Igo, Riccomini, Brun- 
ing, and Pope (2006) found that students with learning disabilities preferred 
copying and pasting notes from web sources. They found that typing or 
handwriting notes was distracting to the learning process and that the best 
learning gains came from studying the notes after identifying key informa¬ 
tion to be learned. They also discovered that writing or typing notes “verba¬ 
tim” resulted in more shallow processing of information than paraphrasing 
main ideas into their own words. Copying and pasting notes to review also 
got students past the concerns of spelling and grammar and handwriting 
readability that can be distracting for them when taking notes. This process 
also decreased study time overall, adding an element of motivation for use. 
Research has also uncovered a number of challenges students face with 
online learning as they transition from a largely teacher-led learning envi¬ 
ronment into the self-directed learning role within the online environment. 
Some students are choosing online learning for credit recovery or because 
they were “unable or unwilling to function in a more traditional classroom 
environment,” which is the “exact opposite of the population that research 
says tends to thrive in the distance environment” (Huett, Moller, Foshay 
& Coleman, 2008, p. 64). Winters, Greene, and Costich (2008) found that 
while students may “view support tools as aiding their self-directed learn¬ 
ing, they did not always use them,” and students have a “poor calibration 
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between what they think they do and what they actually do” (p. 430). A 
second outcome from their study supported the use of mixed-methods re¬ 
search design to gain a more “accurate measure” of students’ self-directed 
learning activities than using just one type of data. Mistler-Jackson and 
Butler-Songer (2000) studied curricular features offered by Internet tech¬ 
nologies for learning science and reported that students are not “spending 
much of their time on understanding or evaluating the information they re¬ 
trieve (higher order thinking skills), but instead on finding quick answers 
and making shallow interpretations of their findings” (p. 2). When teach¬ 
ers scaffolded content into three phases (introductory, development of ex¬ 
pertise, and sharing), students achieved content learning benefits, increased 
self-efficacy, and increased motivation to spend time on task. Butcher and 
Sumner (2011) found that students are not usually successful in using meta- 
cognitive strategies on their own, especially when learning in an online 
environment: “highly contextualized and personal support, like prompting 
from a human tutor - can significantly increase students’ success in regulat¬ 
ing their learning strategies such as monitoring progress towards learning 
goals, and coordinating information from (multiple) sources” (p. 10). 

In order to better understand the individual student experience of learn¬ 
ing online and the extent to which these types of factors impact individual 
student learning success, researchers need more insight into how students 
are personalizing their own learning experience through their choice and use 
of resources in the online curriculum. One way to gain this insight is to “vir¬ 
tually” observe students completing online lessons and collect data on stu¬ 
dent use of resources and thinking strategies for choosing and using lesson 
resources for learning. Collecting both qualitative and quantitative data pro¬ 
vides a more complete picture of student resource use and helps researchers 
better determine the effectiveness of resource use and thinking strategies in 
choosing resources for learning. With a more complete picture of resource 
use and its impact on student learning outcomes, it becomes clear where 
gaps exist for students learning online, so that more effective learning inter¬ 
ventions can be planned and implemented. 

METHODS 


Overview 

A convergent parallel mixed methods research design was used (Cre- 
swell & Plano-Clark, 2011) where qualitative and quantitative data were 
collected in parallel, analyzed separately, and then merged for interpretation 
and discussion. Data were collected in April and May 2013 from students 
before the end of the semester. The quantitative data collected included: 
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1. A list of asynchronous resources used by each student in each lesson; 

2. end-of-lesson assessment scores; 

3. prior semester Science grades; 

4. the student’s primary learning style (category) as measured using the 
Visual, Aural, Read/Write, and Kinesthetic (VARK) Learning Styles 
Inventory, which was a tool that was recommended for teachers to use 
as a part of this particular virtual school program (Fleming, 2012); and 

5. survey items where students rated themselves on learning-related 
traits (pre-study). 

The qualitative data included answers to open-ended questions on a pre¬ 
study survey about their decision making process as they completed a les¬ 
son, and explanations of resource use during and following lesson observa¬ 
tions. In the interpretation stage, the quantitative measures were compared 
with qualitative data to create a more complete picture of how lesson re¬ 
sources were used by students and to determine if there were patterns for 
how students were using the resources to achieve successful (or unsuccess¬ 
ful) learning of lesson objectives. An additional goal was to directly com¬ 
pare the findings from the two types of data for corroboration and validation 
purposes. 

Research Questions 

Mixed Methods Research Question: 

1. To what extent do the qualitative results converge with the quantita¬ 
tive residts to enhance understanding of the student use of asynchro¬ 
nous resources? 

Qualitative Research Questions: 

1. How were students using the asynchronous resources in each lesson? 

2. How did students know when they had completed their learning and 
were ready for the end-of-lesson assessment? 

3. How did students describe their learning process? 

Quantitative Research Questions: 

1. To what extent were asynchronous lesson resources being used by 
students in each lesson? 

2. To what extent did use of lesson resources correlate with the student’s 
performance on the end-of-lesson assessment? 

3. To what extent did the use of resources correlate with the Science 
grade from 1st semester? To what extent did students with different 
grades in Science utilize the resources differently? 

4. To what extent did student use of resources correlate with student 
learning style, as measured by the Student Learning Styles (VARK) 
survey at the beginning of the year? 
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Setting 

This study took place in the 2012-13 school year in an online science 
course. The study included eleven students from the course. Students had 
their own login and password and could access the web-based program cur¬ 
riculum at any time, from any place. Students who started at the beginning 
of the year and stayed on-track with their lessons had four to five total les¬ 
sons per day. At approximately one attendance hour per lesson, plus 30 min¬ 
utes of required PE activity time - students spent approximately five to six 
hours per day on a complete day’s learning. Though the program recom¬ 
mended that students have access to a Learning Coach (usually a parent) 
throughout each learning day to establish and support a daily learning rou¬ 
tine and to focus student effort on learning and strengthen academic perfor¬ 
mance, seven of the participants worked at home on their own one or more 
days per week while their parents were at work. The student population of 
the online course was considered heterogeneous which, combined with the 
small sample size, was why more descriptive statistics were used to analyze 
the results from this study. 

Participants 

The study began on March 22, 2013, with a student email that described 
the study and a follow-up phone contact regarding interest in participating. 
For students who participated in the study, informed consent procedures 
were followed. Due to the time considerations of eight school weeks re¬ 
maining in the semester at that point, and the fact that the target lessons for 
observation were in mid-Unit 4, (with Unit 5 still needed to be completed 
before the end of the semester), 20 students were selected from a larger pool 
of 47 students who met the following criteria: 

• Whatever grade students earned in science first semester, they were still 
carrying that same letter grade at mid-second semester. It was felt that 
this would be a stronger more valid and reliable representation of the 
“grade-level group” differences in resource use and learning strategy 
use (“above average” students with grades of A/B in Science and “aver¬ 
age to below-average” students with grades of C/D/F in Science) 

• The students had not yet completed the first lesson in the observed les¬ 
son series (Unit 4, Lessons 5-9) 

From the initial identified population of students: four students with a 
grade of “A”, two students with a grade of “B”, one student with a grade 
of “C”, three students with a grade of “D”, and one student with a grade of 
“F” (11 total) were included in the study (seven female, four male). All stu¬ 
dents who began the study completed the study. A maximal variation sam¬ 
pling strategy was chosen because it was hypothesized that students with 
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different grades in science would come to the study with varying degrees of 
success in their first semester of science. This might include having differ¬ 
ent strategies for using the learning resources in the lessons, which would 
provide a more complete and complex picture of diversity of resource use 
across the 7th grade science student population. 

Completing a Typical Lesson in the Online Environment 

To complete a typical science lesson, a student selected the lesson link in 
his/her planner, and a lesson “window” opened up. Each lesson followed the 
same basic format, allowing the student to: 

1. read over some background-building text, objectives, and key words; 

2. click on links to view video clips to build background knowledge, and 
can click on additional links for resources such as graphic organizers; 

3. read and answer two to four higher-order questions to begin connect¬ 
ing ideas from the lesson or making deeper level connections, which 
serves as a “self-assessment of learning” tool, followed by a link to 
view the correct answers; and 

4. complete a five to six question multiple choice formative assessment 
(graded but not part of science grade) or a more summative five to 
seven question “quiz” covering material from three to four lessons 
(graded and becomes a part of science grade). 

Nothing about the lesson completion process for students changed while 
students were being observed during this study, except that the student first 
entered the researcher’s Blackboard-Elluminate synchronous lesson room, 
selected the “Share Screen - Desktop” option which opened up a viewing 
window for the researcher to see what the student saw on his/her monitor, 
and then the student clicked on the lesson link and began the lesson. The 
student’s lesson completion progress was observed and recorded by the re¬ 
searcher, and the student was asked to “explain out loud” his/her process 
for going through the lesson. The planned observational script (Appendix 
A) lists the statements that were made during the observation, as a guide for 
the observational process. If a student did not reveal his/her thinking or rea¬ 
soning process during the lesson, the researcher asked questions during the 
lesson and at the end of the observation, according to the planned observa¬ 
tional script. When the student completed the lesson and took the end of les¬ 
son assessment, the researcher asked the planned observational script post¬ 
lesson questions and recorded the answers. Then the student was thanked, 
the next lesson observation day and time was established, and the student 
exited the synchronous lesson room. Following the last lesson observation, 
the researcher asked the end-of-study questions and recorded the answers, 
according to the planned observational script (Appendix A). 
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Data Collection 

Qualitative Data and Validity. Each student completed a “Reflection on 
Learning” survey (Appendix B) in which they rated twenty-seven statements 
and answered six open-ended questions about themselves as online learners 
and their learning process. The survey information provided more specific 
qualitative data on each student’s learning process, capturing specific infor¬ 
mation that would not likely be observed or questioned adequately during the 
observed lessons, as well as student self-ratings on items related to student 
motivation and self-efficacy. Lesson observations and notes were recorded 
and transcribed. Students also answered a series of questions about their use 
of resources during the lesson observation. These statements were organized, 
transcribed, and coded into three overall themes representing the Central 
Phenomenon as the type of metacognitive awareness that students demon¬ 
strated about their own knowledge and thinking (Appendix C). 

Data were validated in the following ways: 

1. The researcher worked with a group of three middle school online sci¬ 
ence teachers from the virtual school program who were familiar with 
the Science curriculum and online student learning processes. They 
agreed to review student statements about their learning, to reach a 
consensual validation of themes. 

2. The researcher took the transcribed qualitative statements from stu¬ 
dents and assembled them for review by the team onto Microsoft Word 
documents. The researcher explained the three levels of metacognitive 
processing that appeared to be indicated from student statements about 
their learning processes. The teachers understood the meaning of and 
differences between the themes. 

3. All statements were reviewed by the team and compared to the group¬ 
ings that had been created by the researcher, based upon the themes. 
One teacher identified two statements in one grouping that should be 
moved to another grouping, and one statement in a second grouping 
that should be moved to another grouping - and all teachers and the re¬ 
searcher came to agreement that the statements made by students were 
at that point grouped in ways that supported the themes of different 
levels of metacognitive processing by students. 

Quantitative Data and Inter-rater Reliability. Student information ob¬ 
tained from the Learning Management System included: end of lesson as¬ 
sessment scores, prior semester Science course grade, and VARK Student 
Learning Style Inventory results. Quantitative data was also collected from 
the 27 “rated” answers from the “learning reflection” student survey. Ad¬ 
ditionally, recorded observations provided quantitative data about the num¬ 
ber and type of resources used per student, and by all students per lesson. 
Virtual school science teachers reviewed 10% of the 55 total recorded 
observations and completed the tracking sheet for resource use in lessons to 
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provide reliability and validity for the tracking sheet used by the researcher 
as a data collection instrument. Inter-rater reliability for the teachers coding 
10% of lessons ranged from 75-81% with an average of 78%. 

Mixed Methods Data. The qualitative themes that emerged from the 
qualitative data indicated that students described three levels of metacogni- 
tive processing or thinking about their learning process and use of lesson re¬ 
sources for learning in the self-directed online environment. As is explained 
in the Results and Discussion section, students indicated that their lesson 
resource use (and self- assessment of readiness for lesson assessment) was 
based on one of the following (Appendix C): 

• A basic awareness that focused on using a resource to gain knowledge 

• A slightly more sophisticated awareness of lesson resource used where 
students indicated specific thinking about how they used the resource to 
gain the knowledge 

• A strategic process for determining which resources would be best to 
use, and how to best use the resource, for learning 

The qualitative data of varying levels of metacognitive thinking strat¬ 
egies was combined with the quantitative data collected to create a more 
complete picture of how students were using the resources to achieve learn¬ 
ing, as well as areas where resources were not being used as effectively for 
learning. In terms of validity and reliability of the overall data collection 
process used, students reported that they were not distracted by the obser¬ 
vation process, that their observed lesson completion process was similar 
to their usual process, and that there was nothing else they needed the re¬ 
searcher to know about their participation in the research study process. 


RESULTS 


Qualitative Data Analysis 

Initially, the themes identified from this small sample of students, who 
were selected for maximal variance, were not clear. In reviewing related re¬ 
search, Flavell (1979) and Shannon (2008) conducted studies that identified 
differences in student thinking and awareness of learning processes similar 
to what was observed in this sample of student data, and their categories of 
metacognitive thinking were organized into themes as follows: 

• Awareness of Knowledge - an understanding of what one does and 
does not know, and what one wants to know 

• Awareness of Thinking - an understanding of cognitive tasks and the 
nature of what is required to complete them 

• Awareness of Thinking Strategies - an understanding of approaches to 
(self) directed learning 
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The above themes were used to address the three qualitative research 
questions in this study: 

1. How are students using the asynchronous resources in each lesson? 

2. How do students know when they have completed their learning and 
are ready for the end-of-lesson assessment? 

3. How do students describe the specific steps of their learning process? 

Table 1 presents data on student statements indicating how they deter¬ 
mined their readiness for lesson assessment. 

Table 1 

Student Self-Determination of Assessment Readiness 


How did you determine you 
were ready to take the 
Assessment? 

Students with 

A/B Grades 

N=29 answers 

Students with 
C/D/F Grades 
N=25 answers 

Identified readiness primarily as getting to 
the assessment in the lesson 

3 (10%) 

2 (8%) 

Identified primarily thoughts/feelings used 
to determine readiness 

7 (24%) 

10 (40%) 

Identified specific steps (strategies) that 
were used to determine readiness or 
feelings of readiness 

19 (66%) 

13 (53%) 


Below are examples of student responses to identifying assessment read¬ 
iness primarily as getting to the assessment in the lesson: 

° “I just go along with the lesson and get as much information as 
possible and then I take the quick check.” 

° “Well I had read everything and I watched all the videos and I did 
the assessment.” 

° “Well I watched all the videos and read most of the book...” 

Below are examples of student responses to identifying assessment read¬ 
iness primarily as thoughts/feelings that were used to determine readiness: 

° “Before I decided I wanted to make sure that I got everything out 
of it and thought about how I was doing.” 

° “.. .1 feel that I’m ready...(couldn’t understand)” 

° “I thought about it but I thought they were easier and I thought I 
had it down. Um, I said pull everything out of your mind and put it 
onto the page.” 

° “Because um I didn’t get a perfect score on my review but I 
thought that I did good, pretty good.” 
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And finally, the examples below are ones that students identify as spe¬ 
cific steps (strategies) that were used to determine readiness or feeling of 
readiness: 

° “Um, well after gathering all the info that I learned, and reading 
at the text and looking at all the examples, 1 knew it so I thought 1 
was ready. 1 did know this was a quiz. Used Crossword puzzle b/c 
thought it was good to review the vocab and understand the quiz 
better and questions.” 

° “I thought I was ready to do it b/c of all the notes 1 took and I cop¬ 
ied down and I thought it would be easy - got a good score on the 
BPOP quiz - and review Q’s - so safe to go on to the assessment.” 

° “I had gone through all the reading I had to do, I did all the videos, 
1 did the review thing I thought 1 had an idea of what was going on 
in the lesson.” 

° “When I got most of the answers right on my Brainpop quiz.” 

Table 2 presents steps that students reported using for their learning, as 
represented by their answers from the pre-study survey. Each row represents 
an open-ended question asked in the survey, and the columns represent the 
answers given by students (grouped when possible) and are represented 
by dividing students into two categories based upon their grade in Science 
(A/B and C/D/F). 


Table 2 

Student Reported Data from the Reflection of Learning Survey 


Student Journey of Learning 

Students with Grade of A, B 

Step in the 
Process 

Students with Grades 
of C, D, F 

3/6 eat breakfast 

Getting Themselves 
Ready for Learning 

3/5 eat breakfast 

2/6 review past lessons 

1/5 review the upcoming lesson 

2/6 review the upcoming lesson 

2/5 just get started on the lesson 

3/6 just get started on the work 

2/5 do something unrelated (www.lumosity.com, 
listen to music, get comfortable) 
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Table 2, Continued 


Student Journey of Learning 

Students with Grade of A, B 

Step in the 
Process 

Students with Grades 
of C, D, F 

2/6 write key words & objectives 

Completing a Typical 
Lesson 


1/6 reads through the lesson 

1/5 write key words and objectives 

2/6 take notes 

4/5 use most of the resources/links 

2/6 complete all lesson activities 

3/5 watch videos 

1/6 “1 just do it” 

1/5 reads the book 

2/6 watch videos 

1/5 goes back and uses the resources 

1/6 use (interactive) gizmos 

1/5 uses review quizzes and activities 

2/6 review 


4/6 finish lesson & take assessment 


2/6 when done with lesson 

How students know 
when they are ready 
to take the Lesson 
Assessment 


1/6 “feel confident” 

2/5 “feel confident, ready” 

4/6 “feel they understand/know" 

2/5 understand/learned the material 

3/6 review lesson to self-assess 
understanding 

3/5 use review/scores 

1/6 uses review score 


1/6 “feel disappointed” 

What students 
do when their 
Assessment Score 
is not as high as 
Expected 

1/5 “gets sad” because 1 have high expectations 
for myself at the beginning of the lesson 

2/6 review for understanding 

1/6 works harder 


3/6 “redo/ask for a redo” 

4/5 use teacher comments to redo/ask for redo 

1/6 “if it’s above a ‘C’ 1 just leave it” 
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Table 2, Continued 


No children are distracting me 



1 feel 1 can pay attention to what I'm 
doing, therefore 1 am doing better 



1 love to read and be on a comptuter 
and online learning is both of those 


1 get motivated by my parents 

1 study hard 

What students report 
makes them a 

1 follow the rules, learn efficiently, and 1 love 
learning 

My parents have high expectations - 
they expect 100% ‘A’ 

Successful Online 
Learner 

1 read webmails, go to the Message Board, and 
attend synchronous lessons 

Use teacher comments to guide work 


1 go over the lessons and try to get my best 
grade - if 1 don’t, 1 try to redo the lesson. 

Go to synchronous lessons before 
unit exams 



Successfully complete lessons and 
remember what I'm learning 



4/6 work or study harder 


4/5 review/study 

3/6 try to bring grade up 


2/5 go back and use the book 

1/6 reviews for understanding 

What students do 
when Learning is not 
successful for Them 

1/5 reads the book pages and (watches) videos 
“over and over” 

1/6 asks Learning Coach for help 

1/5 starts taking detailed notes 

1/6 explains it’s being bored or tired 


1/5 asks Learning Coach for help 


It was anticipated at the outset of the study that students who earned 
grades of A and B in their science course may be using lesson resources 
or learning processes differently than students who were earning grades of 
C, D, and F in science. At first glance, both groups of students appeared 
to be using similar processes throughout each stage of the learning process. 
However, upon closer inspection, A/B students are engaging in more ac¬ 
tivities to recall prior learning and set expectations for what they would be 
learning before they begin the lesson. They also identified a wider variety 
of resource use within lessons, and they indicated use of resources that led 
to spending more time with the curriculum concepts during learning (vir¬ 
tual manipulative activities, note-taking, reviewing learning) which may 
have led to better learning and remembering of concepts. When assessment 
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scores were lower than expected, A/B students reported more instances of 
“working harder” and “reviewing for understand what was missed.” This 
could be interpreted as attempts to correct “incorrectly learned” materi¬ 
al from the lesson. These were examples of skills indicated at the higher 
“Awareness of Thinking” and “Awareness of Thinking Strategies” metacog- 
nitive levels. Students with grades of C/D/F reported resource use as “using 
most of the activities,” but this was not consistent with what was observed 
in the lessons. Students with grades of C/D/F in science were using 40-50% 
of resources overall. When learning did not go well for them, they were 
more likely to just “redo it” than to identify a need to go back and review 
the material for understanding. Students in the C/D/F group also reported 
going back to use the book, taking notes, and reading the book and watch¬ 
ing videos “over and over” - which may indicate less effective metacogni- 
tive strategy awareness and use in the initial learning attempt. 

Quantitative Data Analysis 

The quantitative data collected included student observations of resource 
use, student grades in science, student end-of-lesson assessment scores, and 
student learning style. Analyses of the quantitative data were completed in 
order to answer the research questions which are used below to organize the 
results. 

1. To what extent are asynchronous lesson resources being used by stu¬ 
dents in each lesson? 

Table 3 presents the frequency of resource use from the lesson observa¬ 
tions. 

A majority of students were using the media-rich science videos, which 
demonstrated and explained key lesson concepts and vocabulary, and were 
using the self-check and review activities to self-assess their learning before 
taking the lesson assessment. Many of the activities in the second row of the 
table were activities that engaged students in making a personal connection 
to lesson material and in spending more time working with key concepts 
and constructing learning through taking notes and using tables to compare 
and contrast learning. Use of these resources was related to students with 
higher grades in science and higher end-of-lesson assessment scores. 

1. To what extent does the use of resources correlate with the science 
grade from 1st semester? To what extent are the students who have 
different grades utilizing the resources differently? 

Students with science grades of “A” used an average of 70% of available 
resources across all five observed lessons (Table 4). 
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Table 3 

Frequency of Resources Used by Students across all Observed Lessons 


Online Curriculum Resources Used by Students 

Used by 0-3 Students 

Objectives 

Create a Graphic Organizer 

Make it Personal 

Complete a practice lab activity 

Venn Diagram 

Extra Questions 

Additional Vocabulary 

Virtual Explore Activity 

Study the Figure... 


Used by 4-7 Students 

Read Text Pages 

Brainstorming Activity 

Text Questions 

Copy and Complete Table 

Video Review Quiz 

Key Words 

Describe a Time... 

Virtual Manipulation Activity 

Graphic Organizer 

Crossword Puzzle 

Used by 8-11 Students 

Watch a video 

Review Self-Check 

Narrated slideshow 

Review Quizzes 


Students with grades lower than an “A” used 40-51% of available lesson 
resources for learning. Within these general trends, there was one “A” stu¬ 
dent who used an average of 60% of resources (much lower than other “A” 
students), and one “D” student who used an average of 67% resources (much 
higher than other students with grades below “A”). In using the observation 
data to explain this difference, the “A” student reported (and was observed) 
using a very structured, consistent approach to watching videos and stopping 
them to take notes and reading/taking notes, and getting structured support 
for his learning processes from his learning coach. The “D” student reported 
(and was observed) using many resources and taking notes but stated that she 
did not always understand what she was learning after using them, and did 
not appear to have much learning coach support. According to Cavanaugh 
and Blomeyer’s (2007) components of effective online learning, holding all 
other components equal (curriculum design and student characteristics), this 
may be an example where instructional support factors may have an impact 
in helping students become more effective learners through direct instruction 
of learning strategies for choosing and using lesson resources for learning. 

3. To what extent does use of lesson resources correlate with the stu¬ 
dent’s performance on the end-of-lesson assessment? 

The relationship between use of lesson resources and end-of-lesson as¬ 
sessment scores varied in largely the same pattern as the student’s science 
grade. The specific lesson resources that showed a relationship to end-of- 
lesson score were also the same as those that showed a relationship to stu¬ 
dent grade in science. 

















352 


Schafer-Mayse 


Table 4 

Percent (%) Use of Learning Resources per Student (by Grades in Science) 


Student 
in Study 

Student 

Grade 

in 

Science 

Lesson 

5 

Lesson 

6 

Lesson 

7 

Lesson 

8 

Lesson 

9 

Avg. % 
Resources 
used per 
student 

Avg. % 
Resources 
used by 
Science 
Grade 

1 

A 

73 

85 

80 

86 

82 

81 

(A) 70% 
average 
resources 
used 

2 

A 

67 

69 

60 

71 

73 

68 

3 

A 

67 

46 

77 

64 

45 

60 

4 

A 

73 

77 

73 

79 

55 

71 

5 

B 

27 

46 

53 

64 

36 

45 

(B) 41% 
resources 
used 

6 

B 

10 

31 

40 

43 

27 

36 

7 

C 

33 

46 

47 

57 

36 

44 

(C) 44% 
resources 

used 

8 

D 

67 

54 

53 

57 

36 

53 

(D) 51% 
resources 

used 

9 

D 

87 

77 

60 

57 

55 

67 

10 

D 

33 

31 

20 

43 

36 

33 

11 

F 

33 

15 

77 

50 

27 

40 

(F) 40% 
resources 

used 


• Students with “A” grades averaged 87% end-of-lesson scores and 70% 
resource use. 

• Students with “B” grades averaged 75% end-of-lesson scores and 41% 
resource use. 

• Students “C”, “D”, and “F” grades averaged 62-66% end-of-lesson 
scores and 40-51% resource use. 

4. To what extent does student use of resources correlate with student 
learning style, as measured by the Student Learning Styles (VARK) 
survey taken at the beginning of the year? 

• Students with a Visual learning style comprised four of the six students 
with grades of “A” or “B” in science (Read-Write learners three of six) 
and were using more resources on average. 

° Resources used by this group included reading the text and taking 
notes. These resources provide more opportunities to work with con¬ 
cepts, construct understanding, making connections to prior learned 
information. 
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• Students with a Kinesthetic learning style comprised four of the five 
students with grades of “C”, “D”, or “F” in Science and resource use 
varied between high use and lowest use. These students were observed 
using more “connect/engage” activities, but not using them effective¬ 
ly. Sometimes a student just opened a resource, tried a few things, and 
closed it stating that they did not understand how to use it and partici¬ 
pated in “skimming of the text” but not reading the text or taking notes. 

° These students were also going outside the curriculum to define 
words and get explanations of concepts (in 10 out of 25 lesson ob¬ 
servations) and the words used in the definition were not consistent 
with the terms from the lesson or the assessment. 


DISCUSSION 

In order to be successful as self-directed learners in the online learning 
environment, students need to develop the skills to self-monitor their own 
learning progress while also managing their learning environment and the 
use of resources within that environment (Shannon, 2008). From the results 
on the student survey and observed lesson completion, most students indi¬ 
cated some level of awareness of their learning and what steps they could 
take to improve learning, though all students may benefit from direct in¬ 
struction in strategies to help them become more effective self-directed 
learners. 

Use of Resources 

Overall, students using more resources for their learning earned higher 
end-of-lesson assessment scores and were able to identify and explain more 
specific examples of what they just learned when asked following their les¬ 
son completion. This supports recent learning sciences research regarding 
the impact of effective learning related to making connections between new 
information being learned with prior learned information, spending enough 
time with concepts and skills in working memory to strengthen the learn¬ 
ing connections, and having the motivation and goal orientation to persist in 
learning activities until learning is attained (Shannon, 2008; Project 2016, 
2012). Along with skills in using resources for learning, these students also 
demonstrated more specific strategies when it came to using lesson resourc¬ 
es to self-assess learning progress and determining when more learning was 
needed or when they were ready to take the assessment. Using an average 
of 70% of lesson resources was related to higher end-of-lesson assessment 
scores and higher grades in science, which was consistent with end-of- 
lesson assessment scores being one component of a student’s grade in sci¬ 
ence. It was not clear whether using more resources kept students engaged 
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in learning longer (long enough to make learning “stick”) or because stu¬ 
dents with higher grades were more internally motivated to persist in their 
learning efforts. In explaining their use of resources, all students were able 
to identify a reason for using or not using specific resources, though some 
reasons were more sophisticated and strategic than others. Further research 
into helping students understand the relationship between working with les¬ 
son concepts and targeted use of lesson resources to support learning may 
strengthen student success with self-directed use of lesson resources for 
learning in the online environment. 

Learning Styles or Strategies Effect 

There appeared to be a “Learning Styles” effect. It is not clear, though, 
whether this difference is attributable to student learning styles, or to the 
use of learning strategies. For this study, learning styles were addressed be¬ 
cause one of the diagnostic tools available to teachers to help personalize 
student learning at this virtual school program is the VARK Learning Styles 
Inventory. Thus, it was of interest to see if there was a relationship between 
the learning style identified for a student, using this tool, and his/her use of 
curriculum resources in lessons. 

The relationship observed in the study was that “Read-Write” learn¬ 
ers used more curriculum resources and earned higher grades compared to 
“Kinesthetic” learners. Kinesthetic style learners reported not always under¬ 
standing how to use the resource for learning, and earned lower grades on 
end-of-lesson assessments. Specifically, the A/B/C students with a “Read- 
Write” learning style used more of the resources in the curriculum for their 
learning including reading the text and watching both science videos which 
presented multiple levels of learning details. Several of these students also 
took notes during the video. This provided many opportunities to work with 
the concepts in working memory in ways where they were “constructing” 
their learning of the material and connecting the current concepts to con¬ 
cepts already learned. Each of these components may have provided a cu¬ 
mulative effect to support learning at higher levels for these students. Con¬ 
versely, there were more “Kinesthetic” style learners represented among the 
C/D/F students. This group of students was using fewer resources within the 
curriculum for learning (including not reading the text, not taking notes, and 
not watching the second science video because the concepts were confus¬ 
ing). Though there were interactive resources, they were not often selected. 
Not knowing how to use resources effectively for learning, they were using 
resources that were sometimes equally not helpful for learning lesson infor¬ 
mation, and the words used were not consistent with what was in the lesson 
or the assessment, putting these students at a greater disadvantage for learn¬ 
ing success. 
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Continued research is needed to better understand how students are 
using resources in the online learning environment and to assess levels of 
engagement with online curriculum. With a better sense of how students 
are interacting with the online curriculum, we can begin assessing the ef¬ 
fectiveness of various learning support interventions, measure differences in 
synchronous and asynchronous teacher support for learning, and look at the 
impact of peer learning activities on student learning success. As we get a 
better sense of how students are learning in the online environment, we can 
begin to strategize on how to better personalize student learning support. 

Along with gathering more quantitative data, teachers and curriculum de¬ 
signers can benefit from observing students learning in the online environ¬ 
ment and from asking questions to learn more about how students are us¬ 
ing course resources, making decisions about their learning, and determin¬ 
ing their readiness for assessment. Curriculum designers can benefit from 
first-hand knowledge of how students are using the resources placed into the 
curriculum for learning, and what additional support tools might be needed 
to guide students in how to use resources more effectively. Involving “end 
users” (teachers and students) in this iterative development process can help 
bridge the gap between curriculum designers and curriculum users in how 
the curriculum is used for learning, and strengthen our ability to support vir¬ 
tual learning. 
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APPENDIX A 

PLANNED OBSERVATION SCRIPT PROTOCOL 

Planned Observation Script for use during Observation of Lessons 

Welcome & Introduction I will use before the Lesson begins : 

Good morning/aftemoon. 


Thank you for coming to the Livelesson room to work on your 
science lesson today. As I told you in the letter that you read 
about this research project, I am interested in seeing how stu¬ 
dents are using the learning resources in the curriculum. 


So basically all you are going to do is go through the lesson 
like you usually would, but also “think aloud” about what you 
are doing, so I can understand your thinking and choices. For 
example, when you read the information on the pages of the 
lesson, 1 would like you to read them out loud. If you click 
on a resource and decide not to use it, let me know why you 
didn’t use it. It is important to try to do this lesson just like 
you would if you were doing any other lesson — try not to see 
me as a teacher, but just a research observer, learning how 7th 
grade science students learn online. 


There is no “right” or “wrong” way to do this, and you are not 
earning any kind of grade for this - so please don’t be nervous. 
There are many learning resources available to students within 
each of the lessons in the courses, and I am trying to under¬ 
stand which ones are used by my students in 7th grade science 
- and how they are used in the learning process. I may ask 
you some questions as you go along, but I will try not to inter¬ 
rupt the flow of your learning. For example, I may ask you 
why you didn’t use a particular learning resource, just to un¬ 
derstand your thinking. Just answer with whatever your think¬ 
ing is, about the question. 


I will be recording this session, so that I can go back over it 
for my research notes. Remember, all of this information will 
be kept confidential, and will not be used as a part of your sci¬ 
ence course. Are you ready to start? 
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(I will either say “Good, okay then click on the lesson link and you can 
begin”...or answer questions that may be posed by the student first). 

Sample Questions that may be asked by me during the Lesson 

• What are you learning in this lesson? 

• I noticed you clicked on (that resource) and didn’t use it - can you tell 
me why? 

• You downloaded the graphic organizer to fill out during the video - did 
you print it and fill it in, or fill it out online? Can you tell me about us¬ 
ing it? 

• How did you decide which resources to use and which not to use in this 
lesson? 

• How did you know you were ready to take the assessment? 

After the lesson - and the student has completed the “quick check” 
assessment, I may ask: 

• Did you have the resources you needed for this lesson? Was there any¬ 
thing else that would have been helpful to you? 

• Is there anything else you sometimes use or do that would have been 
helpful in learning today’s lesson (such as watching the pre-recorded 
LL video, found in the Science Teacher Message Board, “googling” in¬ 
formation about lesson content) 

• Why do you think you got the assessment score you did? Is it what you 
thought you would get? 

After the last lesson (the last of the series of 5 lessons for the study) is 
completed - but before I let the student go, I will ask these questions: 

• As you were doing these lessons in the Livelesson room with me ob¬ 
serving, were you able to complete the learning you needed for each 
lesson (not distracted or too nervous)? 

• Was the process you used for these lessons pretty close to what you did 
for all your lessons, or did you find yourself doing things differently be¬ 
cause I was observing? Can you explain what you mean by that? 

• Let’s go to the Science Teacher Message Board - can you tell me what 
areas you accessed on this board throughout the semester? 

0 What resources did you use in that (those) areas? 

° Were they helpful? Would something else have been more helpful? 
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• Is there anything else you want me to know about your participation in 
this study? 

I want to thank you for your time this semester, it has really 
been helpful to me in learning how the science resources were 
used by students this semester — and how we can perhaps im¬ 
prove them for next year. Have a great rest of the semester! 


APPENDIX B 

REFLECTION ON LEARNING STUDENT SURVEY 

The following statements are to help you describe yourself as you see 
yourself as a learner in this program. Please respond to them as if you were 
describing yourself to yourself. Do not omit any item. Please read each 
statement carefully, then select one of the three responses to the right of the 
item. Thanks for your ratings to these statements. 


Statement 

Mostly 

False 

Partly False 
Partly True 

Mostly 

True 

1 like to read 

1 

2 

3 

1 have ability to find the answers to questions 

1 

2 

3 

1 would enjoy an opportunity to learn new skills 

1 

2 

3 

1 do poorly on tests 

1 

2 

3 

Learning is easy for me 

1 

2 

3 

1 enjoy learning with other people 

1 

2 

3 

Learning gives me a feeling of accomplishment 

1 

2 

3 

New subjects scare me 

1 

2 

3 

1 do not think well 

1 

2 

3 

Math has practical applications for me 

1 

2 

3 

Studying is a waste of time 

1 

2 

3 

1 am confident of my ability to work with numbers 

1 

2 

3 

1 hate to read 

1 

2 

3 

Changing teachers concerns me 

1 

2 

3 

1 have confidence in finding the answers to 
questions 

1 

2 

3 
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APPENDIX B, CONTINUED 


Statement 

Mostly 

False 

Partly False 
Partly True 

Mostly 

True 

1 have good skills in using the computer for 
learning 

1 

2 

3 

1 have good communication skills 

1 

2 

3 

1 am confident of my ability to communicate ideas 

1 

2 

3 

1 am confident of my computer skills 

1 

2 

3 

1 feel at ease in the way 1 learn 

1 

2 

3 

Most people get ahead faster than 1 do 

1 

2 

3 

1 find it difficult to learn new skills 

1 

2 

3 

Writing is usually a trying experience 

1 

2 

3 

The people at this school like me 

1 

2 

3 

1 have trouble learning 

1 

2 

3 

1 am confident in using effective strategies 

1 

2 

3 

1 am confident in my reading ability 

1 

2 

3 


REFLECTIVE QUESTIONS 

Directions: Imagine that you are explaining some things about yourself 
to a friend - please describe to them the answers to these questions, as you 
would share with a friend: 

1. As an online student, before I start my lessons I get myself ready for 
learning by... 

2. While working on an online lesson, this is what I do to complete a 
typical lesson... 

3. Describe to your friend how you know when you have completed a 
lesson, and are ready to take the assessment: 

4. When you get your assessment score at the end of the lesson, describe 
what you do if your score isn’t as high as you expected: 

5. Describe what makes you a successful online learner: 

6. Describe what you are doing when your online learning is not success¬ 
ful for you: 
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APPENDIX C 

CENTRAL PHENOMENON AND THEMES FROM QUALITATIVE DATA 

Central Phenomenon - as student statements were reviewed, there was a 
qualitative difference in the level of thinking and strategy use demonstrated 
as they made decisions for which lesson resources to use and why, and in 
their decisions about readiness for the end-of-lesson assessment. 

This phenomenon was further divided into the following Themes: 

• “Awareness of Knowledge” Level of Understanding - which is a level 
of understanding of what one does and does not know, and what one 
wants to know. A student at this metacognitive awareness level is fo¬ 
cused on the material to be learned, and not as focused on the processes 
for learning the material. 

• “Awareness of Thinking” Level of Understanding - which describes a 
level of understanding of cognitive tasks and the nature of what is re¬ 
quired to complete them for learning. A student at this level demon¬ 
strates some awareness of thinking about his or her own learning strate¬ 
gies and how they may or may not be effective. 

• “Awareness of Thinking Strategies” Level of Understanding - which 
describes a level of understanding of approaches to (self) directed 
learning used by students for learning. A student at this level demon¬ 
strates an awareness of several strategies for learning and the ability to 
adjust strategies to achieve different learning purposes or if the student 
self-assesses that more learning is needed. 

Examples of Student explanations related to these themes as they apply 
to use of lesson resources for learning: 

a. Opening Paragraph (real world application of the overall lesson con¬ 
cept) and Lesson Objectives are used “in their head” by students to 
orient themselves to what is in the lesson, and what they will need to 
learn 

b. Key Words are defined using video and text resources, either in notes 
or “in their head” during the lesson and are by understood by students 
as terms to be learned during the lesson 

c. Think/Connect/Engage Activities - are used by a few students (mostly 
in their head but a couple of students do write in their notes) to acti¬ 
vate prior learning and make a personal connection to what is about to 
be learned because it helps them remember 

d. BrainPOP videos are viewed by all students to understand lesson key 
words and concepts, for entertainment value as a learning resource, 
and for their visual demonstration and auditory explanation of how 
processes of Science work (i.e. changes of state, sublimation). 
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e. Discovery Education videos - are viewed by a smaller % of students 
(they were reported by many students as often being harder to learn 
from because not all vocabulary and processes were explained, and 
ended up being more confusing to some students who chose to skip 
them) and are viewed to see processes demonstrated for visual sup¬ 
port and auditory explanations. 

f. Teachlets (custom-created videos) - are viewed for learning informa¬ 
tion, for the ability to pause and take notes, and for the built-in learn¬ 
ing checks 

g. Exploration & Gizmo (virtual manipulation) Activities - are used to 
choose different combinations of steps in a process and see the re¬ 
sults, to try things “in a safe environment”, and because they help stu¬ 
dents learn better by showing the processes of the concepts. For stu¬ 
dents who chose to use it, it they reported it as an engaging activity. 

h. Graphic Organizers, Diagrams, Figures, and Charts - No graphic or¬ 
ganizers were observed being downloaded and used to write notes, 
but students who used them in their head did report looking it over 
to see what the questions are, using the information in their head as a 
guide to what to look for in the videos, and as a review for what in¬ 
formation has been learned. Venn Diagrams, Figures, and Charts were 
used (mostly in their head) to understand the differences and similari¬ 
ties of the concepts being compared and key ideas. 

i. Taking Notes - for students who took notes on paper or e-notes, they 
reported writing a title/date of lesson, defining key words, writing 
down different things in organized ways (using titles/subtitles), and 
gave examples of what was in their notes (words and processes de¬ 
fined, main idea of each paragraph, charts/diagrams copied and filled 
out, text questions answered, review questions and answers recorded). 
A few students reported reviewing notes before the end of lesson as¬ 
sessment and before tests. 

j. Crossword Puzzle Quiz Review - used by a few students to reinforce 
knowledge from previous lessons and review the vocabulary and defi¬ 
nitions before the (3-lesson cumulative) quiz 

k. BrainPOP Quizzes and Lesson Review Questions & Answers - were 
used by many students to self-asses their understanding of the mate¬ 
rial and their readiness for the end-of-level assessment. 

l. Reading the E-Text or Textbook - for several students reading the text 
and taking notes was part of their learning routine, for a few students 
it was only used if the material was harder for them - and then it was 
read or skimmed only, with no notes taken. 
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m. Text Questions - for several students writing down the Q/A was part 
of their learning routine; for other students who reported using them, 
they answered the questions in their head. 

Student reasons given for not using a lesson resource also indicate all 
three levels of metacognitive awareness: 

a. lack of awareness that the resource was in the lesson 

b. technical challenges with using the resource 

c. lack of understanding of how to use the resource (effectively) for 
learning 

d. previous attempts at using the resource just weren’t helpful for learn¬ 
ing 

e. student doesn’t enjoy the resource/activity (talcing notes, using ex¬ 
plore activities) 

f. for one optional practice lab that was assigned, students didn’t have 
the supplies to do the lab and didn’t have any advance notice to obtain 
them for the lesson 



